[elk]

因为不想使用logstash  想偷懒使用filebeat 且新版的filebeat支持grok

先创个一个json文件

{

    "description": "Test pipeline",

    "processors": [{

      "grok": {

        "field": "message",

        "patterns": ["%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"]

}

}

]

}

curl -H 'Content-Type: application/json' -XPUT 'http://10.6.11.176:9200/_ingest/pipeline/Test-pipeline' [email protected]

curl -XPUT '10.6.11.176:9200/myindex/type/1?pipeline=Test-pipeline&pretty' -H 'Content-Type: application/json' -d' {   "message" : "55.3.244.1 GET /index.html 15824 0.043" } '

es里的数据

这边如果想用grok去处理非json的nginx的日志  可以尝试一下

{

    "description": "nginx pipeline",

    "processors": [{

      "grok": {

        "field": "message",

        "patterns": ["%{TIMESTAMP_ISO8601:timestamp} %{TIMESTAMP_ISO8601:time}  %{IP:remote_addr} %{USER:remote_user} %{NUMBER:body_bytes_sent} %{NUMBER:request_time} %{NUMBER:status} %{IP:host} %{URIPATHPARAM:request} %{WORD:request_method} %{URIPATHPARAM:url} %{URIPATHPARAM:http_referrer} %{NUMBER:body_bytes_sent} %{URIPATHPARAM:http_x_forwarded_for} %{URIPATHPARAM:http_user_agent} "]

}

}

]

}

参考

https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ingest-node.html

http://www.axiaoxin.com/article/236/

https://www.felayman.com/articles/2017/11/24/1511527532643.html#b3_solo_h4_9

https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html